Method of detecting blocking artefacts

ABSTRACT

The invention relates to a method of detecting blocking artefacts within a sequence of digital images. The method comprises a step of high-pass filtering a portion of a digital image, intended to supply at least one card of discontinuity pixels. It also comprises a step of detecting blocking artefacts, intended to detect a first type (p 1 ) of elementary blocking artefact and a second type (p 2 ) of elementary blocking artefact from the at least one card of discontinuity pixels. The method finally comprises a step of correcting the blocking artefacts in accordance with their type (p 1,  p 2 ).

The invention relates to a method of processing data corresponding topixels of a sequence of digital images so as to detect a gridcorresponding to blocking artefacts, said method comprising a step ofhigh-pass filtering a portion of a digital image, intended to supply atleast one card of discontinuity pixels, and a step of detecting blockingartefacts from the at least one card of discontinuity pixels.

The invention also relates to a television receiver comprising aprocessing device for implementing the data processing method accordingto the invention.

It notably finds its application in the detection of blocking artefactswithin a digital image which has been previously encoded and thendecoded in accordance with a block-based encoding technique, forexample, the MPEG standard (“Motion Pictures Expert Group”), and in thecorrection of data comprised in these blocks in order to attenuate thevisual artefacts caused by the block-based encoding technique.

The blocking artefacts constitute a crucial problem for the block-basedencoding techniques using a discrete transform of the discrete cosinetransform DCT type. They appear in the form of block mosaics which aresometimes extremely visible in the decoded image sequences. Theseartefacts are due to a strong quantization subsequent to the discretetransform, which strong quantization causes strong discontinuities toappear at the borders of the encoding blocks.

The article entitled “Optimal JPEG Decoding” by J. Jung, M. Antonini, M.Barlaud, Proc. Of ICIP '98, vol. 1, pp. 410-414, Chicago, October 1998describes a data processing method with which blocking artefacts can bedetected and corrected. To this end, said method comprises a step offrequency transform of the wavelet transform type, which is appliedhorizontally and vertically to an image. The result of this transformcomprises two sub-images which have high-frequency coefficients. Thesehigh-frequency coefficients correspond to blocking artefacts or tonatural contours. In accordance with the prior-art method, thehigh-frequency coefficients corresponding to blocking artefacts arespatially located on a grid of 8×8 pixels and have a value which issmaller than a threshold, a value higher than this thresholdcorresponding to a natural contour.

However, this method is only capable of effecting a basic modeling ofthe blocking artefacts, which limits its possibilities of detecting saidartefacts. Moreover, it only searches the blocking artefacts in 8×8pixel grids. The grid may be distorted within the image because of aresampling of the image. This distortion may sometimes be known inadvance, as in the case of the 3/4 encoding format where the width ofthe grid varies in accordance with the 10-11-11 pattern. However, in themajority of cases, this variation is arbitrary because it originates,for example, from a rate transcoding, an image format conversion in a16/9 television receiver, from a 4/3 format into, for example, a 16/9format a zoom in a portion of the image, an AD conversion, or even acombination of these different conversions. In this case, the prior-artmethod only detects blocking artefacts in a grid having a fixed size andposition and applies a post-processing step based on this detection,with the risk of a partial or even inefficient correction.

It is an object of the invention to propose a data processing methodwhich is more efficient.

To this end, the data processing method according to the invention ischaracterized in that the detection step is also intended to detect asecond type of elementary blocking artefact from the at least one cardof discontinuity pixels.

The invention uses the observations illustrated in FIG. 1, representingthe evolution of the luminance Y as a function of several consecutivepixels. In accordance with these observations, two types of blockingartefact profiles p1 and p2 are principally encountered in the imageswhich have been encoded and subsequently decoded in accordance with ablock-based encoding technique. The first profile p1 corresponds to astandard blocking artefact whereas the second profile p2 corresponds toa blocking artefact which is present in an image that has been subjectedto a resampling operation or to an equivalent operation. In the spatialdomain, the first profile p1 is a single step of a staircase whereas thesecond profile p2 is a double step of a staircase.

The method according to the invention also takes the second blockingartefact profile into account by virtue of a more powerful analysis. Themodeling thus effected takes a possible resampling operation of theimage into account, so that the result obtained in the matter ofdetecting blocking artefacts is improved. The blocking artefacts mayalso be detected independently in any grid, thus rendering theprocessing method more efficient both for detecting and for correctingblocking artefacts.

These and other aspects of the invention are apparent from and will beelucidated, by way of non-limitative example, with reference to theembodiment(s) described hereinafter.

In the drawings:

FIG. 1 illustrates, in the spatial domain, the two artefact profiles p1and p2 which are principally encountered in the images encoded inaccordance with a block-based encoding technique,

FIG. 2 is a diagram showing the data processing method according to theinvention,

FIG. 3 is a diagram showing a wavelet transform,

FIG. 4 illustrates the two artefact profiles p1 and p2 in the frequencydomain after a wavelet transform,

FIG. 5 illustrates the location of a blocking artefact as a function ofan artefact profile p1 represented in the frequency domain after wavelettransform,

FIG. 6 illustrates the two artefact profiles p1 and p2 in the frequencydomain after processing by a gradient filter,

FIG. 7 describes a method of correcting blocking artefacts, and

FIG. 8 describes the principle of correcting a blocking artefact of thep2 type.

The invention relates to a method of processing a sequence of digitalimages encoded and decoded in accordance with a block-based encodingtechnique. In our example, the encoding technique used is the MPEGstandard based on the discrete cosine transform DCT, but mayalternatively be any other equivalent standard, such as, for example,the H.263 or H.26L standard. It should be noted that this method mayalso be applied to a fixed image, encoded, for example, in accordancewith the JPEG standard. The processing method first relates to thedetection of blocking artefacts due to these block-based encodingtechniques and subsequently to the ensuing application such as, forexample, post-processing techniques or image quality measurements.

FIG. 2 shows diagrammatically the processing method according to theinvention. Such a method first comprises a step of high-pass filteringFIL (110) a portion of a digital image. This portion is, for example,one of the two fields of a frame if the image is constituted by twointerlaced frames. The filter is applied horizontally and vertically,row by row, to pixels of luminance Y(m,n) of the field of a digitalimage of the sequence, where m and n are integers between 1 and M andbetween 1 and N, respectively, corresponding to the position of thepixel in the field in accordance with a vertical and horizontal axis,respectively, (M=288 and N=720 in, for example, a 576×720 encodingformat).

In the preferred embodiment, the filtering operation is of the wavelettransform type. The wavelet transform, described with reference to FIG.3, is a signal processing technique which consists of a decomposition ofthe image into a plurality of sub-bands, a sub-band comprising filteredimages of smaller resolution. In our example, the wavelet transform usesa bi-orthogonal decomposition. Such a decomposition has the advantage,on the one hand, that a clear differentiation of the contours by virtueof a high-pass filter is effected and, on the other hand, a smoothing ofthe image by virtue of a low-pass filter is effected. For obtaining afirst sub-band level, the wavelet transform comprises the steps of:

-   -   low-pass filtering LP with a filter 1 p 1 followed by a step of        sub-sampling DS2 by 2 along a horizontal direction of the image        I1, so as to obtain an image Ii of the texture which is        sub-sampled in the horizontal direction, with the low-pass        filter 1 p 1=[0.02674875967204570800; −0.01686411909759044600;        −0.07822325080633163500; 0.26686409115791321000;        0.60294902324676514000; 0.26686409115791321000;        −0.07822325080633163500; −0.01686411909759044600;        0.02674875967204570800] proposed by Antonini et al. in the        article “Image Coding Using Wavelet Transform” IEEE Trans. Image        Processing, vol. 1, no. 2, pp. 205-220, April 1992;    -   high-pass filtering HP with a filter hp1 followed by a step of        sub-sampling DS2 by 2 along a horizontal direction of the image        I1 so as to detect the discontinuities in the horizontal        direction and to obtain a discontinuity image Ei which is        sub-sampled in the horizontal direction, with the high-pass        filter hp1=[0.045635882765054703, −0.028771763667464256,        −0.2956358790397644, 0.5574351615905762, −0.2956358790397644,        −0.028771763667464256, 0.045635882765054703] proposed by        Antonini et al.;    -   for each of the two sub-sampled images in the horizontal        direction Ii and Ei, a low-pass filtering step LP with a filter        1 p 1 followed by a step of sub-sampling DS2 by 2 long a        vertical direction so as to obtain a sub-sampled image I2 or E2        h in the vertical direction, respectively;    -   for each of the two sub-sampled images in the horizontal        direction Ii and Ei, a high-pass filtering step BP with the        filter hp1 followed by a step of sub-sampling DS2 by 2 along a        vertical direction so as to obtain a discontinuity image E2 v or        E2 d sub-sampled in the vertical direction, respectively.

The result is an approximation image I2 which has a resolution dividedby 2 and three detail images E2 v, E2 h, E2 d which give the errorsbetween the original image and the approximate image. The detail imagesE2 h and E2 v represent the discontinuities in the horizontal andvertical directions, respectively.

The method also comprises a step of determining the discontinuitycorresponding to blocking artefacts BAD (120). Said step is based onforming thresholds and comparisons between a current filteredcoefficient and filtered coefficients which are adjacent thereto. FIG. 4illustrates the two artefact profiles p1 and p2 as well as theirrepresentation in the frequency domain: W₁(m,k) as a function of k, kbeing an integer representing the position of a pixel in the row m, thisafter wavelet transform such as described hereinbefore.

The method of determining blocking artefacts is described below for thedetection of vertical blocking artefacts by considering W₁ ^(V)(m,k); itis applicable to the detection of the horizontal blocking artefacts byconsidering W₁ ^(H)(k,n).

A vertical artefact corresponding to the profile p1 is detected if thefollowing cumulative conditions are satisfied, W₁ ^(V)(m,n) being acoefficient of the sub-sampled image E2 v:S 1<|W ₁ ^(V)(m,n)<S 2|W ₁ ^(V)(m,n)|>A 1·|W ₁ ^(V)(m,n±1)||W ₁ ^(V)(m,n±2)|<S 3A vertical artefact corresponding to the profile p2 is detected if thefollowing cumulative conditions are fulfilled:S 1<|W ₁ ^(V)(m,n)|<S 2 and S 1<|W ₁ ^(V)(m,n+1)|<S 2|W ₁ ^(V)(m,n)|>A 1·|W ₁ ^(V)(m,n−1)||W ₁ ^(V)(m,n+1)|>A 1·|W ₁ ^(V)(m,n+2)||W ₁ ^(V)(m,n−2)|<S 3 and |W ₁ ^(V)(m,n+3)|<S 3A1 is a predetermined coefficient equal to 4 in our example. S1 and S2are first and second predetermined thresholds, the first thresholdcorresponding to a visibility threshold, the second threshold to thelimit from which the pixel with position (m,n) corresponds to a naturalcontour. They are equal to 2 and 10, respectively, in our example. S3 isa third threshold obtained from the representation in the frequencydomain after wavelet transform of the blocking artefact profiles. In ourexample, it is equal to 1 and serves to make the detection more reliableby controlling the contrast of the discontinuity. It may be particularlyadvantageous in the case of MPEG4 applications, where access to videodata streams and thus to field quantization steps is possible, to varythe thresholds S1 and S2 as a function of said quantization step so asto further improve the efficiency of the processing method. For example,the threshold values are a linear function of the quantization step.

Because of the sub-sampling by 2 of the decomposition in wavelets, thelocation at the approximate pixel of the blocking artefact is not aneasy matter. Indeed, a coefficient of the frequency domain of the firstsub-band may be associated with two pixels in the spatial domain. Thisis why a finer analysis is necessary, taking into account waveletcoefficient signs W₁ ^(V). FIG. 5 shows that a border of the blocksituated between a pixel p(m,2n−1) and p(m,2n), on the one hand, and aborder of the block situated between a pixel p(m,2n) and p(m,2n+1), onthe other hand, correspond to a similar profile in the frequency domain,with the exception of signs. Because of the sub-sampling, thesub-sampled pixels p(m,2n−3), p(m,2n−1) and p(m,2n+1) being representedby a cross, the signs of the transformed coefficients W₁ ^(V)(m,n−1), W₁^(V)(m,n) and W₁ ^(v)(m,n+1) corresponding to said pixels are (+,−,−)for a block border situated between 2 pixels p(m,2n) and p(m,2n+1), and(+,+,−), respectively, for a block border situated between 2 pixelsp(m,2n−1) and p(m,2n) for a discontinuity in the spatial domain having arising edge. If the discontinuity has a falling edge, the signs of thetransformed coefficients W₁ ^(V)(m,n−1) and W₁ ^(V)(m,n) correspondingto the sub-sampled pixels p(m,2n−3), p(m,2n−1) and p(m,2n+1) are (−,+,+)and (−,−,+), respectively, in the two preceding cases. In summary, ifthe signs of the transformed coefficients W₁ ^(V)(m,n) and W₁^(V)(m,n+1) are identical, then the block border is situated between apixel p(m,2n) and p(m,2n+1); if the signs of the transformedcoefficients W₁ ^(V)(m,n−1) and W₁ ^(V)(m,n) are identical, then theblock border is situated between a pixel p(m,2n−1) and p(m,2n). Blockingartefacts may be localized for the artefacts having a profile of thetype p2 in accordance with a similar principle.

In a particularly advantageous embodiment, the filtering operation is agradient filtering operation using the filter hp2=[1,−1,−4,8,−4,−1,1].This filter is applied horizontally and vertically, row by row, to theluminance pixels Y(m,n) of the field of a digital image of the sequence.The result of this filtering operation is preferably constituted by twocards of discontinuity pixels, a horizontal card Eh and a vertical cardEv. As the majority of resampling operations is performed in thehorizontal direction, the horizontal card Eh showing the verticaldiscontinuities may suffice in a first approximation. However, theprocessing method according to the invention will have an optimalefficiency when it is based on processing the two cards of discontinuitypixels. Other gradient filters are possible such as, for example, thehigh-pass filter of the wavelet transform hp1 proposed by Antonini etal. The filter hp2 is particularly easy to implement and reliablyapproximates the filter hp2.

FIG. 6 illustrates the two artefact profiles p1 and p2 in the spatialdomain, as well as their representation in the frequency domain afterfiltering with the filter hp1 or hp2. In the frequency domain, the firstprofile p1 corresponds to a peak, whereas the second profile p2corresponds to a double peak.

In this case, the step of determining discontinuities corresponding toblocking artefacts comprises a sub-step of detecting natural contoursand non-visible artefacts. To this end, coefficient values filteredhorizontally Yfh(m,n) and/or vertically Yfv(m,n) must be between thefirst and second thresholds S1 and S2 so as to be able to correspond toa blocking artefact. The condition is preferably taken for the absolutevalue of coefficients filtered as follows:S 1<|Yfh(m,n)|<S 2 and S 1<|Yfv(m,n)|<S 2As an alternative, the following condition is used:S′1<|Yfh(m,n)|² +|S′1 Yfv(m,n)|² <S′2,in which S′1 and S′2 have the same function as S1 and S2.

The threshold values depend on the filter used. For the filter hp1, wetake, for example S′1=0.6 and S′2=400, S1=0.5 and S2=20.

The step of determining the discontinuities corresponding to blockingartefacts comprises a sub-step of detecting blocking artefacts. Avertical artefact corresponding to the profile p1 is detected byscanning the field in a horizontal direction corresponding to the row mif the following condition is satisfied:|Yfv(m,n)|>|Yfv(m,n+k)| with k=−2, −1, +1, +2.The border of the block is localized between the pixel of position (m,n)and that of position (m,n+1) if |Y(m,n)−Y(m,n−1)|<|Y(m,n)−Y(m,n+1)| andbetween the pixel of position (m,n−1) and that of position (m,n) in theopposite case.

An artefact corresponding to profile p2 is detected if the followingcumulative conditions are satisfied:f 1·|Yfv(m,n)|<(|Yfv(m,n−1)|+|Yfv(m,n+1)|)|Yfv(m,n−1)|>f 2·|Yfv(m,n−2)||Yfv(m,n+1)|>f 2·|Yfv(m,n+2)|with f1=6 and f2=2 in the preferred embodiment.

The border of the block is localized between the pixel of position(m,n−1) and that of position (m,1). The detection of a horizontalartefact corresponding to each profile p1 and p2 is effected in asimilar manner by scanning the horizontal card Eh comprising thecoefficients Yfh(m,n) filtered in a vertical direction corresponding tothe column n. The step of determining the discontinuities which has beendescribed hereinbefore has the advantage that it is particularly easy toimplement.

A first application of the data processing method according to theinvention is constituted by the MPEG detection, i.e. the detection of asequence of digital images that have been encoded and subsequentlydecoded in accordance with the MPEG standard and of its grid of blockingartefacts among analog image sequences. This MPEG detection is effectedat the level of a television receiver and is generally followed by astep of post-processing images, intended to correct said blockingartefacts which are present in the grid.

To this end, the processing method also comprises a step of selectingSEL (130) segments in a horizontal row or a vertical row of the field,which segments comprise a number of consecutive discontinuity pixelswhich is higher than a fourth predetermined threshold S0. Indeed, theisolated discontinuities generally correspond to a supplementary noise,while the blocking artefacts which are due to a coarse quantization ofthe DCT coefficients generally cause linear faults to appear along theencoding blocks. The value S0 of the predetermined threshold must not betoo low so as not to favor the false detections. It must neither be toohigh so as not to constrain the selection too much by reducing thenumber of segments of detected elementary artefacts. In practice, thevalue S0 is fixed at 3 for a field of 288 rows of 720 pixels.

Advantageously, the processing method also comprises a step ofsearching, within the field, a set of grid rows, a grid row having adensity of elementary block effects present in the segments which issubstantially larger than that of its neighboring rows. Such a stepallows an even further reduction of the risk of false detections.

A second application of the data processing method according to theinvention is constituted by post-processing images intended, to correctthe blocking artefacts which are present in a grid. Said grid has beendetermined by the method described previously or is known as, forexample, the post-processing operation is effected in an MPEG-4 videodecoder. The correction depends on the profile of the detected blockingartefact.

If the blocking artefact corresponds to the profile p1, the correctiondescribed with reference to FIG. 7 is applied. The method of correctingblocking artefacts comprises the steps of

-   -   computing a first discrete cosine transform DCT1 (71) of a first        set of N data u situated at the left or above the border of the        block;    -   computing a second discrete cosine transform DCT1 (72) of a        second set of N data v situated at the right or below the border        of the block and adjacent to the first set;    -   computing a global discrete cosine transform DCT2 (73) of a set        of 2N data w corresponding to the concatenation CON (70) of the        first and second sets and providing a set of transformed data W;    -   determining PRED (74) a predicted maximum frequency kwpred from        the transformed data U and V obtained from the first (71) and        second (72) transform DCT1, computed in the following manner:        kwpred=2.max(kamax, kvmax)+2        with        kumax=max(kε{0, . . . ,N−1}/abs(U(k))>T),        kvmax=max(kε{0, . . . ,N−1}/abs(V(k))>T),        where T is a threshold which is different from zero;    -   correcting ZER (75) by setting the odd transformed data W from        the global discrete cosine transform to zero, whose frequency is        higher than the predicted maximum frequency, yielding corrected        data W′;    -   computing an inverse discrete cosine transform IDCT2 (76) of        corrected data, yielding filtered data w′ which are subsequently        intended to be displayed on the screen.

If the blocking artefact corresponds to the profile p2, the correctionmust be modified considerably. Indeed, the position of the border of theblock must be given more precisely because of the double step of thestaircase corresponding to the profile p2, as illustrated in FIG. 8.This is why the correction method preliminarily comprises a step ofreadjusting the luminance value of the intermediate pixel p(n) intendedto give said luminance value the luminance value of the pixel which issituated directly on its right p(n+1). The steps described hereinbeforeare then applied, with the border of the block being situated at theleft of the intermediate pixel, which then forms part of the segment v.

It is alternatively possible to cause the luminance value of theintermediate pixel to correspond to that of the pixel on the left, or tothat of the pixel having the nearest luminance value. In both cases, thepositioning of the segments u and v is adapted accordingly so as toapply the correction step.

A third application of the data processing method according to theinvention is constituted by measuring the block level of the field fromblocking artefacts which are present in the grid so as to determine thequality of the images. The quality measurement may be effected at thelevel of a television receiver in which the grid has been determined bythe method previously described or at the level of an MPEG-4 videodecoder, with the grid already being known so as to ensure a givenservice quality.

The level of the block B of the field f is preferably obtained bysumming the amplitudes of the filtered values W₁ ^(V)(mn) correspondingto elementary blocking artefacts, i.e.${B(f)} = {\sum\limits_{m,n}\quad\left\lfloor {{{\delta\left( {\left( {m,n} \right) \in {artV}} \right)}{\kappa\left( {m,n} \right)}{{W_{1}^{V}\left( {m,n} \right)}}} + {{\delta\left( {\left( {m,n} \right) \in {artH}} \right)}{\kappa\left( {m,n} \right)}{{W_{1}^{H}\left( {m,n} \right)}}}} \right\rfloor}$wherein δ(x)=1 if x is true and 0 if not, artV and artH comprising thepixels detected as blocking artefacts.

Such a measurement has the advantage that it takes the amplitude W₁ ofthe degradation into account. It also takes into account the position(m,n) of the degradation, while a weighting coefficient K(m,n) as afunction of the perception of the human visual system can be introduced.Moreover, this measurement allows determination of a block level for agrid having an arbitrary dimension or even being variable with respectto time.

It is possible to implement the processing method according to theinvention by means of a television receiver circuit, said circuit beingsuitably programmed. A computer program stored in a programming memorymay cause the circuit to perform the different operations describedhereinbefore with reference to FIG. 2. The computer program may also beloaded into the programming memory for reading a data carrier such as,for example, a disc comprising said program. The reading operation mayalso be performed by means of a communication network such as, forexample, the Internet. In this case, the service provider will put thecomputer program in the form of a downloadable signal at the disposal ofthose interested.

Any reference sign between parentheses in the present text should not beconstrued as being limitative. Use of the verb “comprise” and itsconjugations does not exclude the presence of elements or steps otherthan those stated in the claims. Use of the article “a” or “an”preceding an element or step does not exclude the presence of aplurality. of such elements or steps.

1. A method of processing data corresponding to pixels of a sequence ofdigital images so as to detect blocking artefacts, said methodcomprising a step of high-pass filtering (110) a portion of a digitalimage, intended to supply at least one card of discontinuity pixels, anda step of detecting (120) blocking artefacts, intended to detect a firsttype (p1) of elementary blocking artefact from the at least one card ofdiscontinuity pixels, said method being characterized in that thedetection step is also intended to detect a second type (p2) ofelementary blocking artefact from the at least one card of discontinuitypixels.
 2. A data processing method as claimed in claim 1, wherein thehigh-pass filtering step (110) is suitable for using a wavelettransform, intended to supply at least one card of sub-sampleddiscontinuity pixels corresponding to a sub-band of said transform.
 3. Adata processing method as claimed in claim 2, also comprising a step oflocalizing the elementary blocking artefact in a row of the portion ofthe digital image from signs of filtered values of the sub-sampledpixels of the card of discontinuity pixels.
 4. A data processing methodas claimed in claim 1, wherein the high-pass filtering step is intendedto supply two cards of discontinuity pixels, one horizontal card and onevertical card.
 5. A data processing method as claimed in claim 1,wherein the high-pass filtering step is suitable for using a gradientfilter.
 6. A data processing method as claimed in claim 1, comprising astep of measuring the image quality, intended to provide a blockingartefact level from filtered values of the pixels corresponding to theblocking artefacts.
 7. A data processing method as claimed in claim 1,comprising a step of correcting the blocking artefacts in accordancewith their type (p1, p2).
 8. A television receiver comprising aprocessing device using the data processing method as claimed in claim7, suitable for detecting blocking artefacts within a sequence ofdigital images and for correcting them with a view to displaying asequence of corrected digital images on a screen of said receiver.
 9. Avideo decoder suitable for providing a sequence of decoded digitalimages and comprising a processing device using the processing method asclaimed in claim 7, suitable for detecting blocking artefacts within thesequence of decoded digital images and for correcting them so as toprovide a sequence of corrected digital images.
 10. A computer programproduct comprising a set of instructions which, when loaded into acircuit, cause said circuit to perform the method of processing digitalimages as claimed in claim 1.